Teaching Machines to Code: Neural Markup Generation with Visual Attention

We present a deep recurrent neural network model with soft visual attention that learns to generate LTEX markup of real-world math formulas from their images. Applying neural sequence generation techniques that have been very successful in the fields of machine translation and image/handwriting/speech captioning, recognition, transcription and synthesis; we construct an image-tomarkup model that learns to produce syntactically and semantically correct LTEX markup code of over 150 words long and achieves a BLEU score of 89%; the best reported so far for the Im2Latex problem100. We also visually demonstrate that the model learns to scan the image left-right / up-down much as a human would read it.